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Abstract.  To  address  the  data  review  burden  and  improve  the  reliability  of  the  ultrasonic  inspection  of  large  composite 
structures,  automated  data  analysis  (ADA)  algorithms  have  been  developed  to  make  calls  on  indications  that  satisfy  the 
detection  criteria  and  minimize  false  calls.  The  original  design  followed  standard  procedures  for  analyzing  signals  for 
time-of- flight  indications  and  backwall  amplitude  dropout.  However,  certain  complex  panels  with  varying  shape,  ply  drops 
and  the  presence  of  bonds  can  complicate  this  interpretation  process.  In  this  paper,  enhancements  to  the  automated  data 
analysis  algorithms  are  introduced  to  address  these  challenges.  To  estimate  the  thickness  of  the  part  and  presence  of  bonds 
without  prior  information,  an  algorithm  tracks  potential  backwall  or  bond-line  signals,  and  evaluates  a  combination  of 
spatial,  amplitude,  and  time-of- flight  metrics  to  identify  bonded  sections.  Once  part  boundaries,  thickness  transitions  and 
bonded  regions  are  identified,  feature  extraction  algorithms  are  applied  to  multiple  sets  of  through-thickness  and  backwall 
C-scan  images,  for  evaluation  of  both  first  layer  through  thickness  and  layers  under  bonds.  ADA  processing  results  are 
presented  for  a  variety  of  complex  test  specimens  with  inserted  materials  and  other  test  discontinuities.  Lastly, 
enhancements  to  the  ADA  software  interface  are  presented,  which  improve  the  software  usability  for  final  data  review  by 
the  inspectors  and  support  the  certification  process. 


INTRODUCTION 

The  ultrasonic  inspection  of  aerospace  composites  has  been  demonstrated  to  be  one  of  the  most  effective  methods 
to  detect  critical  defect  types  and  ensure  the  reliability  of  composite  structures  [1-3].  Most  inspection  applications  of 
composites  and  bonded  joints  are  based  on  pulse-echo  ultrasonic  testing  (UT)  and  manual  C-scan  data  interpretation. 
Using  amplitude  and  time-of-flight  C-scan  data,  delaminations,  disbonds,  porosity,  and  foreign  materials  can  be 
detected  and  located  in  depth.  However,  the  ultrasonic  inspection  of  large  composite  structures  requires  significant 
manpower  and  production  time.  To  address  this  inspection  burden  and  ideally  increase  inspection  reliability,  software 
tools  and  automated  data  analysis  (ADA)  algorithms  [4-9]  have  been  developed  to  support  the  assessment  of  ultrasonic 
data  from  composite  components.  Recent  work  on  the  design  of  ADA  algorithms  [7-9]  follows  standard  procedures 
for  analyzing  signals  for  time-of-flight  indications  and  backwall  amplitude  dropout.  However,  certain  complex 
composite  structures  with  varying  shape,  thickness  transitions  and  the  presence  of  bonds  can  greatly  complicate  this 
interpretation  process. 

Most  prior  work  concerning  the  inspection  of  bonded  regions  in  composites  has  focused  on  bond  integrity  [10]. 
The  topic  of  detecting  discontinuities  and  characterizing  complex  composites  parts  with  bonded  components  and  ply 
drops  has  received  some  recent  attention  on  enhanced  ultrasonic  NDE  techniques  [11-13].  An  example  test  panel  with 
a  bonded  pad-up  region  is  shown  in  Fig.  1.  Frequently,  the  edges  of  adhesive  layers  are  found  at  thickness  transitions, 
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producing  multiple  shifting  signals  and  coherent  noise  in  the  pulse-echo  UT  data,  resulting  in  a  challenging  signal 
interpretation  task.  In  this  paper,  enhancements  to  the  automated  data  analysis  algorithms  are  introduced  to  address 
these  challenges.  One  requirement  is  to  estimate  the  thickness  of  the  part  and  presence  of  bonds  without  prior 
information.  This  task  is  accomplished  by  tracking  potential  backwall  signals,  detecting  the  presence  of  multiple 
signals  and  step  changes  which  indicate  bonded  sections,  and  through  the  application  of  smart  spatial  filters  for 
estimating  the  panel  thickness  and  additional  bonded  sections  with  varying  signal  levels.  Once  part  boundaries, 
thickness  transitions  and  bonded  regions  are  identified,  feature  extraction  algorithms  are  applied  to  multiple  sets  of 
through-thickness  and  backwall  C-scan  images,  for  evaluation  of  both  first  layer  through  thickness  and  layers  under 
bonds.  In  this  study,  a  challenge  set  of  test  data  was  selected  to  verify  the  ADA  algorithms  over  a  wide  range  of 
complex  parts  and  artificial  defects  located  both  above  and  below  the  bond-line,  as  shown  in  Fig  1. 


Hitco  Specimen  #13,  Nominal  Cure  Gap  0.200"  *  Double  Gap  0.100"*  Lap  0.100"*  *  Gaps  and  lap  at  -45°  in  ply  12 


Dimensions:  40"  x  14"  Rectangular  inserts  are  1"  x  0.5"  with  1"  spacing  between  each.  Located  at  ply  17. 

(S"  on  right)  Inserts  featured  in  long  leg  were  placed  on  edge  to  be  exposed  when  part  is  trimmed. 


FIGURE  1.  (a)  Diagram  and  (b)  photo  of  Hitco  test  panel  #13  with  a  bonded  pad-up  region  and  inserted  materials. 


AUTOMATED  DATA  ANALYSIS  ALGORITHM  DESIGN  LOR  BONDED  REGIONS 

The  design  objective  for  the  software  is  to  make  calls  on  all  anomalous  indications  that  satisfy  the  inspection 
criteria  while  minimizing  false  calls  associated  with  normal  part  conditions.  Anomalous  indications  studied  in  this 
program  include  inserted  materials,  porosity,  ply  Taps  and  gaps’,  and  wrinkles.  Inserted  materials  may  occur  during 
manufacturing,  possibly  originating  from  such  items  as  backing  paper,  glove  (poly),  breather  or  miscellaneous  factory 
materials.  Other  artificial  materials  like  Grafoil,  FEP  and  ‘sticky’  notes  have  been  used  for  testing  purposes  and  can 
produce  challenging  ultrasonic  inspection  conditions  for  demonstration  tests.  Shapes  of  inserted  materials  may  be 
triangular  (e.g.  corners  of  backing  paper),  rectangular  or  more  irregular  shapes.  A  minimum  dimension  criterion  for 
an  indication  is  typically  defined  with  the  test  specification.  Often,  0.25”  x  0.25”  (0.63  cm  x  0.63  cm)  squares  are 
prescribed  as  minimum  dimensions  for  inserted  materials.  There  are  a  number  of  potential  false  call  features  that 
should  ideally  be  vetted  by  any  automated  data  analysis  algorithm  as  an  operator  would.  Such  part  features  include 
edges,  ply  drops,  panel  radii,  stringers  /  beam-type  structures,  and  noodles  at  radii.  Operators  typically  use  knowledge 
of  the  part  and  a  view  of  the  TOF  C-scan  image  to  distinguish  part  features  from  an  amplitude  drop-out  due  to 
anomalous  conditions.  Lastly,  scanning  artifacts  such  as  streaking,  water  on  the  part,  or  scanner  backlash  may  also 
be  present  in  ultrasonic  data  and  can  be  easily  rejected  by  a  trained  operator. 

The  base  ADA  algorithm  process  steps  were  first  described  in  prior  work  [7-9].  The  proposed  automated  data 
analysis  approach  strives  to  automate  the  entire  ultrasonic  data  analysis  process  including  the  creation  of  the  C-scan 
amplitude  and  TOF  images  to  make  indication  calls.  The  current  design  of  the  ADA  algorithm  does  require  access  to 
the  full  A-scan  data  acquired  for  the  part,  which  might  be  a  limitation  for  some  older  UT  systems.  Building  on  the 
prior  designs,  the  following  steps  will  emphasize  algorithm  updates  to  enhance  detection  capability  of  regions  with 
bonds  and  adjacent  thickness  transitions,  and  minimize  the  false  call  rate. 

(a)  Evaluate  calibration  data.  The  first  step  is  to  evaluate  the  inspection  system  using  calibration  data  acquired 
from  a  known  reference  standard.  A  signal-to-noise  acceptance  criterion  is  evaluated  for  the  back  wall  signal  with 
respect  to  the  internal  noise  echoes  (e.g.  20  dB  SNR)  to  ensure  the  transducers  are  operating  properly.  Call 
performance  on  known  defects  in  the  reference  standard  must  also  be  verified.  One  change  from  prior  work  is  calling 
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‘sub-rejectable’  indications,  under  the  dimension  and  area  criteria.  Some  of  the  inserted  material  indications  in 
standards  do  not  always  meet  the  strict  dimension  criteria;  however,  verification  of  these  indications  is  important. 

(b)  Process  front-wall  signals  and  identify  part  edges.  The  front-wall  (FW)  C-scan  is  used  to  locate  the  part 
domain  and  edges.  By  evaluating  the  edge  locations  and  applying  the  inspection  zone  criterion  for  distance  from 
edges  (for  example,  1 .0  in),  a  map  identifying  the  different  inspection  zones  can  be  created.  The  algorithm  calls  three 
zones:  an  acreage  region  away  from  the  edges  in  the  part,  and  edge  region  based  on  the  edge  criterion,  and  a  thin 
buffer  region  that  is  within  roughly  0.24M  (typically  3  pixels)  of  where  the  FW  signal  drops  off.  A  buffer  region  is 
applied  in  order  to  avoid  false  calls  in  the  edge  region  due  to  a  lack  of  consistent  backwall  signals  at  the  part  edge. 
The  FW  signals  also  provide  a  means  to  check  that  signal  levels  are  adequate,  which  was  found  to  reduce  false  calls. 

(c)  Evaluate  signal  width  for  setting  the  start  and  end  of  gates.  The  front- wall  signals  are  used  to  evaluate  the 
pulse  width  during  inspection.  Since  gates  must  be  automatically  set  by  the  ADA  algorithm  to  produce  internal  and 
backwall  C-scan  images,  it  is  important  to  have  a  precise  measure  of  how  the  signals  rise  and  ring  down  in  time  in 
order  to  avoid  false  calls.  More  details  on  this  process  can  be  found  in  prior  work  [9]. 

(d)  Identifying  bonds  in  test  parts.  Improving  calls  in  bonded  sections  of  production  parts  requires  detecting 
their  location  and  extent.  From  the  perspective  of  pulse-echo  ultrasound,  both  the  bond-line  and  the  backwall  signal 
produce  reflections,  somewhat  smaller  in  magnitude  than  regular  backwall  signals  without  an  adhesive  layer  present. 
The  first  challenge  here  is  distinguishing  regions  with  bonds  from  specimens  with  strong  multiple  reflections  often 
from  thin  panel  regions.  As  well,  signal  from  bond-line  does  not  always  meet  criteria  for  a  backwall  signal,  as  shown 
in  the  TOF  C-scan  image  in  Fig.  2(a).  To  evaluate  the  regions,  signals  are  tracked  before  and  after  the  1st  backwall  / 
bond-line  signals  as  shown  in  Fig.  2(b)  and  Fig.  2(c)  respectively.  This  provides  a  rough  map  of  the  missing  TOF 
signals  in  Fig.  2(a).  Then,  clustering  routines  with  a  minimum  area  criteria  is  applied  to  group  regions  of  multiple 
potential  ‘backwall’  signals.  Next,  two  signal  metrics  are  evaluated  for  each  multi-signal  region,  shown  in  Fig.  3,  to 
distinguish  regions  with  bonds  and  thin  multi-layer  structures.  The  first  metric  (x-axis)  compares  the  ‘time-of-flights’ 
of  multiple  signals  and  normalizes  the  time  difference  with  respect  to  1/2  of  the  second  signal.  Signals  due  to  multiple 
reflections  in  a  single  layer  will  have  a  value  near  0.  The  second  metric  (y-axis)  quantifies  the  change  in  amplitude 
of  signals.  It  has  been  observed  that  bond  signals  and  backwall  signals  under  bonds  are  close  in  amplitude,  versus 
multiple  reflections  in  a  layer  which  exhibit  significant  decay.  Figure  3  presents  results  for  these  two  metrics  for  a 
number  of  examples  of  bond  data  and  thin  layer  composite  data.  A  simple  linear  classifier  can  be  used  to  call  regions 
with  bonds  with  good  reliability. 


FIGURE  2.  C  -scan  images  showing  bond  detection  and  interpretation  process:  (a)  initial  backwall  TOF  C-scan  estimate,  (b) 
TOF  maps  (b)  before  and  (c)  after  estimated  ‘backwall’  signal,  (d)  called  bond  region  (red)  with  include  edge  pixels  (light  blue), 
(e)  revised  backwall  TOF  C-scan  map  with  complete  bond,  (f)  backwall  TOF  C-scan  under  bond,  (g)  C-scan  amplitude  plot  for 
internal  gate  (for  TOF+  calls),  and  (h)  C-scan  amplitude  plot  of  backwall  signal  under  bond  (for  BW2  calls). 
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FIGURE  3.  Comparison  of  time-of-flight  and  amplitude  ratio  metrics  for  a  sample  of  ‘no  bond’  and  ‘with  bond’  regions. 


(e)  Estimate  backwall  and  bond-line  depth  using  enhanced  backwall  search  process.  To  identify  the  features 
in  the  C-scan  amplitude  map,  simultaneous  interpretation  of  the  C-scan  TOF  map  is  needed  to  distinguish  internal  and 
backwall  signals.  Thus,  an  accurate  assessment  of  the  true  backwall  depth  and  depth  of  multiple  layers  due  to  bonds 
over  the  entire  part  is  needed.  In  prior  work,  a  smart  ‘backwall’  search  process  for  the  TOF  C-scan  map  was  introduced 
[9].  Recent  improvements  to  this  process  have  been  made  by  using  current  state-of-the-art  in-painting  algorithms  [14] 
providing  greater  accuracy  in  estimating  the  backwall  level  under  regions  with  defects,  complex  ply  drops,  surface 
curvature  and  general  poor  data  quality  (e.g.  data  dropout).  However,  this  backwall  estimation  process  is  complicated 
by  the  presence  of  bond  regions  with  a  second  ‘true’  backwall  signal.  Ultrasonic  scans  are  presented  in  Fig.  4 
contrasting  the  response  for  inspections  from  the  tool  side  and  bag  side  of  the  panel  #13  pad-up  region  including  the 
bond.  Often,  there  is  an  abrupt  transition  from  the  backwall  (shifting  deeper)  to  where  an  adhesive  layer  is  being 
detected  earlier  in  time,  as  shown  in  Fig  4(c)  and  Fig  4(d).  When  regions  with  bonds  are  present,  the  backwall 
estimation  process  is  broken  into  separate  steps  with  masks,  first  outside,  then  inside  for  each  identified  bonded  region. 
An  iterative  process  was  then  implemented  for  tracking  disconnected  signals  at  the  edge  of  the  adhesive  layers, 
improving  backwall  tracking.  This  process  was  able  to  reduce  false  calls  in  thickness  transitions  neighboring  bonded 
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FIGURE  4.  Ultrasonic  scan  from  (a)  tool  side  and  (b)  bag  side  of  pad-up  region  with  an  adhesive  bond.  Signals  from  the  edge 
of  the  bond-line  signal  in  (c)  and  (d)  can  produce  early  ‘through  thickness’  indications  in  time  in  the  composite  thickness 
transition  zone.  Results  for  the  refined  backwall  and  bond-line  estimates  are  presented  for  the  (e)  tool  side  and  (f)  bag  side. 
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regions.  However,  some  further  refinements  may  still  be  needed.  For  example,  the  edges  of  bonded  regions  are  still 
the  biggest  area  of  false  calls.  One  can  easily  set  up  a  criteria  to  remove  most  of  these  based  on  their  proximity  to  the 
edge  of  an  adhesive  layer  plus  a  thickness  transition.  Much  more  ‘noise’  also  appears  to  be  present  between  the  front- 
wall  and  adhesive  layer.  However,  at  this  time,  it  was  chosen  to  be  more  sensitive  to  ensure  no  significant  defects  are 
missed.  TOF  map  results  for  the  pad-up  region  in  Hitco  panel  #13  using  the  ‘enhanced’  backwall  search  for  the  1st 
level  and  2nd  level  (backwall  signal  under  bonds)  are  shown  in  Fig.  2(e)  and  Fig.  2(f)  respectively. 

(f)  Identify  part  zones  and  transitions.  Due  to  the  presence  of  ramps,  stringers  and  radii  changes  in  composite 
parts,  this  interpretation  task  can  become  complicated.  By  using  the  front  wall  amplitude  C-scan  image  and  the 
estimated  backwall  TOF  map,  the  different  region  types  can  be  classified.  The  following  inspection  zone  categories 
are  identified  using  the  front- wall  /  edge  metric  map:  (0)  off-part,  (1)  acreage  zone  (away  from  edge  or  radii),  (2)  edge 
zone,  (3)  backwall  thickness  transition  zone,  and  (4)  region  with  a  bond.  In  the  ADA  software  interface,  metrics  are 
now  being  presented  to  the  user  for  each  called  indication  whether  that  location  meets  the  criteria  for  an  edge  ( Cedge ), 
a  thickness  transition  ( Ctrans )  and  a  region  with  a  bond  ( Cbond ).  Values  that  exceed  the  criterion  for  each  category 
are  highlighted  in  ‘yellow’.  Examples  are  shown  in  Fig.  5  and  6  with  indications  for  the  three  part  zone  metrics. 

(g)  Create  internal  signal  C-scans  and  evaluate  noise  level  for  part.  A  through  thickness  internal  ‘only’  signal 
C-scan  can  be  generated  following  the  front-wall  signal  and  using  a  backwall  follower  with  the  estimated  backwall 
signal  TOF  map.  Internal  signals  above  a  threshold  level  are  then  called  to  produce  an  internal  indication  C-scan  map 
(TOF+)  with  binary  values.  An  example  around  a  bonded  region  is  shown  in  Fig.  4(g).  Improved  detection  reliability 
and  reduced  false  call  rate  were  achieved  through  adapting  the  base  threshold  level  with  respect  to  the  measured 
internal  noise  level  within  the  gate  [9]. 

(h)  Create  backwall  signal  ‘only’  C-scans.  Using  the  TOF  map  for  the  backwall  signal  with  a  tight  gate,  a 
backwall  signal  ‘only’  amplitude  C-scan  can  be  generated.  Loss  of  backwall  signals  below  a  prescribed  threshold 
level  is  then  called  to  produce  a  backwall  loss  C-scan  map  (BW-)  with  binary  values.  Adaptive  call  criteria  have  also 
been  applied  to  address  sensitivity  to  variation  in  backwall  signal  level  observed  in  the  wide  range  of  test  data  studied 
to  date  [9]. 

(i)  Perform  feature  extraction  and  classification  of  internal  (TOF+)  signals.  Feature  extraction  algorithms 
can  now  be  applied  to  the  internal  indication  C-scan  map  (TOF+).  Contour  finding  algorithms  are  then  applied  and 
clustering  is  used  to  combine  closely  spaced  indications,  and  area  and  dimension  metrics  are  evaluated  for  all 
significant  clusters.  The  location  of  each  cluster  is  assessed  with  respect  to  the  inspection  zone  maps  in  order  to  ensure 
it  is  evaluated  using  the  correct  inspection  zone  criteria.  All  indications  that  satisfy  the  amplitude,  dimension  and  area 
criteria  are  recorded  along  with  the  call  metrics  and  identified  inspection  zone. 

(j)  Perform  initial  feature  extraction  and  classification  of  backwall  signal  loss  (BW-)  and  signal  loss  under 
bonds  (BW2):  The  C-scan  amplitude  plot  for  the  backwall  signals  can  be  gated  to  create  a  binary  map  of  backwall 
signals  below  a  threshold  (BW-)  and  backwall  signals  under  bonds  (BW2).  Again,  contour  finding  algorithms  are 
then  applied  and  clustering  is  used  to  combine  closely  spaced  indications  that  satisfy  the  minimum  area  metric 
requirements.  The  location  of  each  cluster  is  assessed  with  respect  to  the  inspection  zone  maps  in  order  to  ensure  it  is 
evaluated  using  the  correct  criteria.  All  indications  that  satisfy  the  amplitude  and  area  criteria  are  recorded  along  with 
the  call  metrics  and  identified  inspection  zone. 

(k)  Generate  porosity  criteria  map  and  perform  evaluation  (BWp).  To  perform  the  porosity  evaluation,  the 
backwall  C-scan  TOF  map  can  be  transformed  into  an  attenuation  map  for  setting  the  acceptance  criteria  as  a  function 
of  depth.  The  C-scan  amplitude  plot  for  the  backwall  signals  can  then  be  gated  using  the  corresponding  porosity 
criteria  map  threshold  to  create  a  binary  map  with  porosity  indications  (BWp).  A  sweep  algorithm  is  used  to  identify 
regions  of  the  part  that  exceed  the  porosity  area  criteria  and  within  the  region  size  requirement  (for  example,  25% 
pixels  meeting  the  acceptance  criteria  over  a  1 .0  in.  square  region.)  At  this  stage,  a  cluster  search  algorithm  (as  applied 
to  the  TOF+  and  BW-  indication  map)  groups  neighboring  regions  from  the  porosity  (BWp)  indication  map. 

(l)  Evaluate  TOF  thickness  indications  (TOF2).  Often,  delaminations  or  foreign  material  located  in  the  first  few 
plies  can  be  difficult  to  detect  using  either  internal  defect  gating  or  backwall  dropout  metrics.  However,  internal 
time-of-flight  plots  can  often  provide  clear  indications  of  such  features.  The  current  algorithm  applies  a  change  in 
time  threshold  from  the  backwall  to  form  the  indication  map  (TOF2).  The  cluster  search  and  size  evaluation  process 
is  applied  here  to  verify  such  indications  satisfy  the  acceptance  criteria.  Enhancements  to  the  algorithm  have  been 
implemented  to  address  panel-to-panel  thickness  variation  and  thickness  ‘noise’  at  transitions.  For  more  info,  see  [9]. 

(m)  Compile  calls  with  metrics.  Indications  with  call  metrics  are  presented  to  the  operator.  These  include  called 
indication  type  (TOF+,  BW-,  BWp,  TOF2,  BW2),  indication  size  (length,  width,  and  area),  median  signal  level  (in 
dB)  with  respect  to  the  call  criteria,  and  zone  metrics  {Cedge,  Ctrans  and  Cbond).  Progress  has  been  made  to  sort 
calls  based  on  likelihood  of  the  indication  being  a  significant  defect,  based  on  these  metrics  and  grouped  by  call  type. 
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RESULTS  FOR  TEST  PANELS  WITH  BONDED  REGIONS 


An  example  of  ADA  processing  results  is  presented  for  the  test  specimen  #13  as  shown  in  Fig.  1.  This  specimen 
contains  artificial  defects  that  have  been  added  at  varying  locations  and  ply  depth,  including  above  and  below  the 
adhesive  layer.  Diagrams  are  shown  in  Fig.  4  of  the  ultrasonic  scan  of  the  (a)  tool  side  and  (b)  bag  side  of  pad-up 
region  with  the  bond.  Example  ADA  toolkit  interface  results  are  presented  in  Fig.  5  and  Fig.  6  scanned  from  tool  side 
and  bag  side  respectively.  In  the  first  column  of  the  results  windows,  a  code  is  used  to  describe  what  criterion  was 
met  to  make  the  call:  (TOF+)  amplitude  C-scan  of  internal  signals,  (BW-)  C-scan  of  backwall  dropout,  (TOF2)  TOF 
C-scan  internal  signals,  and  (BW2)  C-scan  of  backwall  dropout  under  a  bond.  Indications  are  listed  in  the  spreadsheet 
display  in  the  upper  left  and  corresponding  numbers  are  presented  identifying  the  indications  in  the  C-scan  image 
display  on  the  right.  For  these  ADA  evaluations  for  the  two  different  scan  orientations,  the  three  triangular  inserts  in 
the  bond  region  were  all  correctly  called.  For  the  tool  side  scan  (Fig.  5),  the  left  most  triangle  is  in  front  of  the  bond, 
while  the  right  two  triangle  are  behind  the  bond.  The  left  triangles  were  called  by  TOF2  and  BW2  algorithms,  while 
the  right  triangle  was  only  called  by  BW2.  For  the  bag  side  scan  (Fig.  6),  the  left  most  triangle  is  behind  the  bond, 
while  the  right  two  triangle  are  in  front  of  the  bond.  This  left  most  triangle  was  called  using  both  the  BW2  and  TOF2 
criteria.  Examples  of  the  backwall  signal  loss  maps  under  the  bonds  are  shown  in  Fig  7.  While  the  signal  loss  is 
clearly  observed  with  the  presence  of  the  triangular  inserts,  in  a  few  cases,  the  drop  for  the  right  triangle  in  Fig.  7(b) 
may  not  quite  meet  a  fixed  call  criteria.  Note,  this  indication  was  called  by  the  TOF2  algorithm.  Thus,  there  are 
benefits  of  using  multiple  call  criteria  in  these  bonded  regions.  As  well,  adapting  the  call  criteria  for  BW2  indications 
as  a  function  of  the  bond  and  backwall  signal  quality  would  ensure  more  reliable  calls  for  varying  part  conditions. 

Note,  in  both  Figures  5  and  6,  some  false  indications  are  displayed,  nearly  all  due  to  backwall  loss  indications  at 
edges  and  transition  regions  with  bonds  present  that  satisfy  the  call  criteria.  While  operator  review  can  quickly  reject 
these  indications,  a  key  objective  is  to  improve  the  performance  of  the  ADA  software  for  rejecting  such  indications. 
Refinements  have  been  made  to  minimize  these  calls  over  a  wide  test  set  including  a  number  of  panels  with  bonds 
scanned  from  different  sides  and  varying  gain  levels. 

Lastly,  indications  for  the  six  inserted  materials  at  the  radii  are  also  observed  in  the  TOF  map  in  Fig.  5  and  6. 
However,  the  inserted  ‘sticky’  note  material  was  found  to  have  the  weakest  response  by  far.  Unfortunately,  the  signal 
dimensions  and  area  do  not  meet  the  call  criteria  in  these  two  cases.  It  appears  that  the  paper  material  of  the  ‘sticky’ 
note  is  prone  to  being  saturated  with  resin  during  cure,  and  produces  much  weaker  responses  than  the  other  five 
inserted  test  materials.  Careful  finger-printing  of  ‘sticky’  note  defects  is  needed  to  properly  test  the  ADA  algorithms. 


FIGURE  5.  Example  ADA  toolkit  interface  results  for  Hitco  test  panel  #13,  scanned  from  tool  side,  with  TOF  C-scan  view. 


120006-6 

6 

Distribution  A.  Approved  for  public  release  (PA):  distribution  unlimited. 


‘sticky’  note 
indication 


ftwflPfcnwpY  a-rttrapv  graph  I  *»«  PjnH  frAfejw&OT 


:  S«Bic  !  *  0  S  Q  E  <  fl  «  *» 


three  bond 
region 
indications 


FIGURE  6.  Example  ADA  toolkit  interface  results  for  Hitco  test  panel  #13,  scanned  from  bag  side,  with  TOF  C-scan  view. 


FIGURE  7.  C-scan  of  backwall  loss  under  the  bond-line  with  three  triangular  inserts  for  (a)  tools  side  and  (b)  bag  side  scans. 


SOFTWARE  TOOLS  FOR  DATA  REVIEW  AND  TO  SUPPORT  CERTIFICATION 

A  software  interface  for  the  automated  data  analysis  (ADA)  toolkit  has  been  presented  in  Fig.  5  and  6.  The  ADA 
Toolkit  is  a  component  of  the  open-source  NDI  Toolbox™  led  by  TRI/ Austin.  The  main  view  provides  a  summary  of 
the  found  indications  in  the  analyzed  data,  a  visual  presentation  of  an  indication  map  and  quantitative  metrics  assisting 
the  operator  in  understanding  why  each  call  was  made.  The  software  uses  a  modular  approach  to  designing  and 
transitioning  ADA  algorithms.  The  ADA  Toolkit  ‘plug-ins’  consist  of  a  configuration  file  and  the  analysis  code  that 
can  be  imported  within  the  interface.  Python  is  the  base  programming  language  for  the  ADA  algorithm  code.  The 
ADA  algorithms  have  also  been  parameterized  with  values  stored  in  the  configuration  file  such  that  the  acceptance 
specifications  can  be  modified  within  the  software  for  a  new  part  or  inspection  procedure.  A  configuration  viewer  is 
provided  within  the  user  interface  for  modifying  all  of  the  parameter  settings  for  the  ADA  algorithm.  A  design  goal 
is  to  create  a  tool  that  can  be  easily  adapted  by  expert  operators  following  test  specimen  acceptance  criteria  for  different 
composite  material  types,  thicknesses  and  ideally  part  complexity.  To  support  the  operator  review  of  indications,  the 
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ADA  Toolkit  provides  an  indication  viewer,  to  review  quantitative  metrics,  image  and  signal  data,  and  qualitative  call 
descriptions  for  each  indication.  For  more  information  on  the  software  interface  design,  see  reference  [7-9]. 

Software  features  have  also  been  integrated  into  the  ADA  Toolkit  to  facilitate  algorithm  design  and  parameter 
optimization  studies,  and  support  certification.  The  software  can  run  a  set  of  UT  files  in  batch  mode,  process  the 
results,  compare  them  to  truth  tables  for  each  file,  and  compile  the  total  correct  call,  missed  call,  and  false  call  rates 
for  the  set.  Features  have  been  transitioned  to  import  ‘truth’  tables  and  match  known  flaws  with  the  ADA  generated 
indication  table  results  (.ind  file).  An  indication  comparison  file  can  be  generated  which  organizes  all  of  the  ADA 
called  indications  into  three  groups:  true  positives  (TP),  missed  calls  (MC)  and  false  calls  (FC).  Note,  an  indication 
position  error  factor  is  currently  set  to  0.63  cm  (0.25”),  providing  some  practical  flexibility  in  matching  ADA 
indications  with  the  truth  table.  The  output  report  from  this  process  is  expected  to  be  quite  useful  for  both  parametric 
studies  for  code  optimization,  verifying  performance  with  calibration  panels,  and  quantitatively  demonstrating 
validation  for  certification. 


VERIFICATION  TEST  RESULTS 

The  test  panel  demonstrations  presented  in  prior  work  [8-9]  and  in  previous  sections  for  panels  with  bonds  do 
provide  a  good  first  step  to  support  the  use  of  ADA  algorithms  for  NDE  data  review.  However,  in  order  to  certify  the 
use  of  ADA  algorithms  for  the  review  of  production  ultrasonic  NDE  data  from  composite  inspections,  (1)  the  detection 
capability  for  the  desired  target  size(s),  (2)  the  rate  of  false  calls,  and  (3)  ideally  the  time  difference  for  NDE  inspector 
secondary  review  of  ADA  reported  results  (with  respect  to  making  the  same  calls  through  a  manual  data  review)  must 
be  evaluated  quantitatively.  In  order  to  evaluate  the  true  positive  (TP),  missed  call  (MC)  and  false  call  (FC)  rate  of 
the  ADA  algorithms  with  statistical  rigor,  software  tools  were  developed  to  automatically  perform  the  study. 

A  tedious  but  critical  step  is  the  creation  of  ‘truth’  tables  with  very  accurate  information  for  all  the  flaws  in  each 
scan,  including  dimensions  and  locations.  For  an  initial  test  study,  ‘truth’  tables  (as  .csv  files)  were  created  for  40 
selected  Mistras  and  TRI/ Austin  UT  data  files  from  project  test  panels  and  select  Hitco  calibration  panels.  These  data 
sets  and  truth  tables  contain  over  400  insert  materials  indications  and  natural  indications  associated  with  porosity. 
They  also  include  all  of  the  toughest  test  panel  cases  such  as  saturated  ‘sticky’  note  inserts  and  partially  scanned  inserts 
at  the  radii.  Wrinkles,  laps  and  gaps  have  also  been  included  in  the  truth  table,  but  detection  rates  for  these  flaws  are 
calculated  separate  from  inserts  and  natural  indications  associated  with  porosity.  In  addition  to  the  40  panel  set 
presented  in  prior  work  [9],  a  separate  intermediate  set  of  test  data  was  selected  to  challenge  the  ADA  algorithms  that 
includes  a  wide  range  of  complex  parts  with  bonds  and  artificial  defects  located  both  above  and  below  bond  lines.  It 
is  critical  to  design  a  test  set  that  provides  the  widest  array  of  part  and  flaw  characteristics  that  represent  the  expected 
range  of  data  NDE  inspectors  review  daily  while  also  including  a  statistically  significant  number  of  flaw  indications. 
For  comprehensive  certification,  a  significant  number  of  production  panels  should  be  tested  as  well,  ideally  with  some 
actual  verified  indications. 

Statistical  results  are  presented  in  Table  1  for  the  40  challenge  test  files.  For  this  study,  the  accuracy  target  was  to 
detect  0.25”x0.25”  (0.63  cm  x  0.63  cm)  inserts.  In  a  POD  evaluation,  the  goal  is  to  assess  the  detection  capability  for 
the  accuracy  target.  With  a  scan  resolution  of  0.080”x0.080”  (2  mm  x  2  mm),  such  inserts  are  expected  produce  3 
pixels  in  the  x-  and  y-scan  directions.  Because  of  this  criteria,  the  correct  ’insert’  calls  (TP  =  true  position)  were 
grouped  by  flaw  size  into  the  three  categories:  (1)  TP  >  0.25”sq  (square)  inserts,  (2)  %TP  =  0.25”sq  inserts  and  (3) 
%TP  <  0.25”sq  inserts.  Past  results  from  ref.  [9]  are  included  in  the  first  column  and  new  results  from  recent  revisions 
of  the  ADA  algorithm  are  presented  in  the  next  four  columns.  For  the  baseline  Case  1,  most  of  the  17.6%  misses  for 
0.25”sq  inserts  were  associated  with  the  Nacelle  test  panels,  where  the  pixel  area  criteria  was  not  always  met.  Through 
a  series  of  algorithm  improvements  discussed  in  this  paper,  significant  improvements  were  made  in  terms  of  the  missed 
call  rate  (Case  5):  (1)  TP  >  0.25”sq  =  98.1%,  (2)  %TP  =  0.25”sq  inserts  =  90.2%  and  (3)  %TP  <  0.25”  =  44.4%.  One 
topic  that  was  explored  was  the  effect  of  calling  ‘sub-rejectable’  indication.  For  Case  4,  indications  were  called  having 
only  two  pixels  in  width  in  either  the  dx  or  dy  dimensions  (while  meeting  the  same  area  criteria  of  8  pixels  at  0.080” 
scan  resolution).  Some  improvements  were  achieved  in  detecting  previously  missed  calls  including  a  wrinkle. 
However,  the  false  call  rate  did  increase  by  26%.  When  such  indications  are  presented  to  the  operator  in  the  user 
interface,  a  yellow  highlight  is  applied  to  the  length  or  depth  to  indicate  dimensions  under  the  criteria  (see  Fig.  5  and 
6.)  Note,  four  large  misses  still  remain  in  the  test  set.  These  are  solely  dependent  upon  the  shape  and  area  call  criteria 
not  being  met.  Two  of  the  four  misses  are  due  to  saturated  ‘sticky’  note  triangles,  one  is  due  to  a  weak  poly  triangle 
(that  can  be  called  by  modifying  the  B  W-  call  criteria  parameter)  and  one  is  due  to  an  insert  only  partially  scanned  at 
the  radii  edge.  Calling  sub-rejectable  BW-  and  TOF+  indications  in  signal  level  is  being  considered  to  help. 
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Because  of  the  complexity  of  some  of  the  composite  test  specimens,  there  have  been  challenges  to  reducing  the 
number  of  false  calls  in  prior  work  [9]  as  shown  in  Case  1 .  Significant  progress  has  been  made  to  eliminate  these 
false  calls  through  properly  addressing  the  review  of  ultrasonic  signals  from  bonds,  an  enhanced  backwall  tracking 
algorithm,  taking  care  in  making  calls  when  the  front-wall  signal  is  weak  in  magnitude,  and  through  the  use  of  adaptive 
thresholding  based  on  panel  noise  levels.  As  well,  careful  expert  review  of  all  of  the  indications  was  performed  for 
the  40  panel  set.  Through  this  review  process,  a  number  of  natural  indications  were  discovered  associated  with 
porosity  and  wrinkles.  The  updated  results  presented  in  Case  3  (wrt  Case  1)  show  a  nearly  62%  reduction  in  the  false 
call  rate  for  the  40  panel  set,  while  new  indication  calls  (TP)  are  also  being  made.  By  breaking  out  the  calls  as  shown 
in  Table  1,  the  algorithm  source  of  the  many  of  the  true  positive  and  false  calls  can  be  identified.  Interestingly,  the 
backwall  loss  criteria  only  contributes  to  call  roughly  8.4%  of  the  inserts  but  generates  in  over  60%  of  the  false  calls 
at  thickness  transitions.  One  observation  (separate  from  this  40  panel  study)  from  production  part  data  is  that  BW- 
false  calls  are  extremely  rare  compared  to  this  40  panel  test  set.  Much  of  these  backwall  loss  indications  exist  because 
of  tape  on  the  part  edges,  water  on  the  back  side  of  the  part,  or  the  orientation  of  the  probe  relative  to  the  curved  part 
surface.  False  calls  of  greater  interest  and  challenge  are  associated  with  the  TOF2  calls  at  the  edge  of  bonds  at 
thickness  transitions.  Some  examples  are  found  in  Fig.  5  and  6  and  these  indications  are  also  observed  in  production 
data.  Adapting  the  TOF2  call  criteria  as  a  function  of  location  should  reduce  these  remaining  false  calls  in  Case  5. 


TABLE  1.  Results  for  40  panel  test  set.  Metrics  include  true  positive  (TP)  detections,  false  calls  (FC)  and  missed  calls  (MC). 


Case  1 

Case  2 

Case  3 

Case  4 

Case  5 

Ref.  [9] 

Rev.  7/201 5a 

Rev.  7/201 5a 

Rev.  7/201 5a 

Rev.  7/201 5b 

area  threshold  (#  pixels,  0.08"  scan  res.) 

8 

8 

8 

8 

8 

dimension  threshold  (#  pixels,  0.08"  s.r.) 

3 

3 

3 

2 

2 

thrsh2,  for  BW-  calls  (dB  drop) 

-13.0 

-13.0 

-13.0 

-13.0 

-13.0 

TP 

337 

344 

365 

368 

370 

MC 

49 

42 

37 

34 

32 

FC 

263 

191 

100 

126 

119 

Area  (sq  in.)  per  false  call 

46.7 

64.2 

122.7 

97.4 

103.1 

%FC TOF+ 

14.5% 

20.9% 

4.0% 

6.4% 

8.4% 

%FC  BW- 

59.7% 

41.4% 

62.0% 

57.1% 

60.5% 

%FC  BWp 

0.0% 

0.0% 

0.0% 

0.0% 

0.0% 

%FC TOF2 

25.9% 

30.9% 

34.0% 

36.5% 

31.1% 

%FC BW2 

NA 

6.8% 

0.0% 

0.0% 

0.0% 

MC>0.25"sq  inserts  /  porosity 

4 

5 

6 

4 

4 

MC=0.25"sq  inserts  /  porosity 

25 

20 

14 

14 

13 

MC<0.25"sq  inserts  /  porosity 

13 

13 

10 

10 

10 

%TP>0.25"sq  inserts  /  porosity 

98.1% 

97.6% 

97.1% 

98.1% 

98.1% 

%TP=0.25"sq  inserts  /  porosity 

82.4% 

85.9% 

90.2% 

90.2% 

90.9% 

%TP<0.25"sq  inserts  /  porosity 

27.8% 

27.8% 

44.4% 

44.4% 

44.4% 

%TP TOF+  only 

25.5% 

26.2% 

24.1% 

23.4% 

23.5% 

%TP BW-  only 

13.7% 

9.3% 

8.2% 

8.4% 

8.4% 

%TP TOF2  only 

25.2% 

25.9% 

29.6% 

28.5% 

28.4% 

%TP multiple  calls 

35.6% 

38.7% 

38.1% 

39.7% 

39.7% 

MC,  wrinkles,  laps  and  gaps 

4 

4 

7 

6 

5 

TOTAL,  wrinkles,  laps  and  gaps 

17 

17 

32 

32 

32 

%TP,  wrinkles,  laps  and  gaps 

76.5% 

76.5% 

76.5% 

81.3% 

84.4% 

Expert  Review  of  False  Call  Indications 

S 

V 

S 

CONCLUSIONS  AND  FUTURE  WORK 

In  this  paper,  enhancements  to  the  automated  data  analysis  algorithms  are  introduced  to  address  the  challenges  of 
inspecting  complex  panels  with  varying  shape,  ply  drops  and  the  presence  of  bonds.  One  requirement  was  to  estimate 
the  thickness  of  the  part  and  presence  of  bonds  without  prior  information.  This  task  was  accomplished  through 
tracking  potential  backwall  signals,  detecting  the  presence  of  multiple  signals  and  step  changes  which  are  indicators 
of  bonded  sections,  and  through  the  application  of  smart  spatial  filters  for  estimating  the  panel  thickness  and  additional 
bonded  sections  with  varying  signal  levels.  Once  part  boundaries,  thickness  transitions  and  bonded  regions  are 
identified,  feature  extraction  algorithms  are  applied  to  multiple  sets  of  through-thickness  and  backwall  C-scan  images, 
for  evaluation  of  both  first  layer  through  thickness  and  layers  under  bonds.  A  challenge  set  of  test  data  was  selected 
to  verify  the  ADA  algorithm  performance  over  wide  range  of  complex  parts  and  artificial  defects  located  both  above 
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and  below  bond  lines.  Today,  the  detection  performance  appears  to  satisfy  the  current  target  requirements  for  0.25” 
sq.  inserted  materials  while  the  false  call  rate  is  now  considered  reasonable.  Lastly,  improvements  to  the  ADA  software 
interface  were  presented,  which  improve  the  software  usability  with  the  NDI  operator  in  data  review. 

At  this  stage,  the  certification  process  has  been  defined  [15]  and  is  being  followed  to  ensure  the  performance  of 
the  ADA  software  with  operator  data  review  meets  the  acceptance  criteria.  The  integrated  software  tools  will  be  quite 
valuable  to  certify  the  use  of  ADA  algorithms  for  (1)  the  detection  capability  for  the  desired  target  size,  (2)  the  rate  of 
false  calls,  and  (3)  the  time  for  NDE  inspector  secondary  review  of  ADA  reported  results.  Future  work  is  expected  to 
further  refine  the  evaluation  as  additional  test  data  are  introduced  to  the  software.  Ideally,  the  ADA  algorithm  should 
be  optimized  with  all  three  metrics  in  mind.  The  use  of  prior  part  information  on  panel  thickness  and  bond  location 
is  also  being  considered  which  could  simplify  this  evaluation  process.  Software  tools  have  been  implemented  to 
identify  part  information  from  the  filename  and  directory  encoding.  Initial  demonstrations  have  been  made  where 
part  type  is  identified  and  key  meta-data  like  part  thickness  and  defect  maps  are  stored  and  tracked  over  time. 
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